Skip to content

Fix: 🐛 Zenodo release timeout issue#188

Open
slugb0t wants to merge 2 commits into
stagingfrom
upload-heartbeat
Open

Fix: 🐛 Zenodo release timeout issue#188
slugb0t wants to merge 2 commits into
stagingfrom
upload-heartbeat

Conversation

@slugb0t

@slugb0t slugb0t commented Jul 10, 2026

Copy link
Copy Markdown
Member

Summary by Sourcery

Harden the Zenodo release workflow against long-running network operations and dropped SSE connections, while improving client-side reconciliation of publish status and proxy configuration for streaming responses.

Bug Fixes:

  • Add per-request timeouts to Zenodo and GitHub API calls and upload/download operations to prevent hangs in the publication workflow.
  • Persist Zenodo deposition status as "publishing" and uniformly mark it as "error" on failures so the server-side workflow can be reliably reconciled after connection drops.
  • Introduce SSE heartbeats to keep long-lived Zenodo publish streams alive across proxies and browsers, and clear them when the workflow ends.
  • Handle cases where the Zenodo publish SSE stream ends without a terminal event by polling server status and updating the UI to reflect success, error, or still-processing states.
  • Adjust UI messaging and icons to distinguish pending server-side processing from errors, avoiding false failure indications for long-running releases.

Enhancements:

  • Refactor Zenodo publication error handling into a shared helper to standardize logging, DB updates, and progress event emission.
  • Update deployment proxy config to disable response buffering and extend response timeout to better support streaming progress for long Zenodo releases.

Deployment:

  • Configure kamal proxy response_timeout and disable response buffering to support multi-minute SSE progress streams without premature termination or buffering.

@fairdataihub-bot

Copy link
Copy Markdown

Thank you for submitting this pull request! We appreciate your contribution to the project. Before we can merge it, we need to review the changes you've made to ensure they align with our code standards and meet the requirements of the project. We'll get back to you as soon as we can with feedback. Thanks again!

@sourcery-ai

sourcery-ai Bot commented Jul 10, 2026

Copy link
Copy Markdown

Reviewer's Guide

Introduce network timeouts, heartbeat-based SSE connection keep-alive, and client-side reconciliation/pending UI to prevent Zenodo releases from appearing failed when long-running uploads or proxy drops occur; also refactor server-side publication error handling into a shared helper and tweak deployment proxy config to support long-lived streams.

File-Level Changes

Change Details Files
Add explicit request timeouts for Zenodo API calls, uploads, and GitHub interactions to prevent indefinitely hanging publication workflows.
  • Define separate API and upload timeout constants to reflect expected operation durations.
  • Pass AbortSignal.timeout to all Zenodo metadata/deposition/publish-related fetch requests.
  • Apply longer upload timeout to Zenodo bucket uploads, repository zip downloads, and GitHub release asset downloads.
ui/server/services/archival/zenodo.ts
Refactor Zenodo publication error handling into a reusable helper and adjust DB status management to distinguish in-progress from terminal states.
  • Set zenodoDeposition status to 'publishing' at workflow start and keep it during upsert, instead of resetting to 'draft'.
  • Introduce failPublication helper that logs, updates DB status to 'error', emits SSE error progress events, and returns a standardized failure result.
  • Replace per-step try/catch error handling with calls to failPublication for metadata, upload_metadata, update_repo, upload_files, and publish steps.
ui/server/services/archival/zenodo.ts
Improve client-side handling of Zenodo publish SSE streams by tracking terminal events, reconciling with server status when streams drop, and exposing a new 'pending' UI state.
  • Add reconcilePublishStatus polling helper to query backend zenodo workflow status when SSE ends without a terminal event.
  • Track whether streaming started and whether a terminal step was received; only show immediate error toast when the stream fails before starting, otherwise run reconciliation.
  • Extend modal UI with a 'pending' state, new icon, explanatory copy, and CTA to navigate back to the dashboard when the server is still processing.
ui/pages/dashboard/[owner]/[repo]/release/zenodo.vue
Configure the deployment proxy to support long-lived, unbuffered SSE responses for Zenodo release progress.
  • Set proxy response_timeout to 600 seconds to allow long-running releases.
  • Disable response buffering so SSE progress events are flushed incrementally and heartbeat pings remain effective.
  • Minor whitespace cleanup in deploy.yml proxy/hosts config.
ui/config/deploy.yml
Keep the SSE connection alive during long Zenodo publication steps using periodic heartbeat pings and ensure cleanup on completion.
  • Introduce a 15-second heartbeat interval that writes SSE comment pings while the client is connected and response is not ended.
  • Clear the heartbeat interval in the publication handler finally block to prevent leaks.
  • Leave existing onProgress forwarding and terminal event emission intact so the client can detect completion or error steps.
ui/server/api/[owner]/[repo]/release/zenodo/index.post.ts

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@fairdataihub-bot

Copy link
Copy Markdown

Thanks for making updates to your pull request. Our team will take a look and provide feedback as soon as possible. Please wait for any GitHub Actions to complete before editing your pull request. If you have any additional questions or concerns, feel free to let us know. Thank you for your contributions!

@what-the-diff

what-the-diff Bot commented Jul 10, 2026

Copy link
Copy Markdown

PR Summary

  • Improvements in handling live connections in deploy config
    The deploy.yml configuration file has been updated to better handle long live connections.

  • More reliable Zenodo publication status
    A new function, reconcilePublishStatus, has been added to manage Zenodo publication status especially when streams suddenly drop.

  • Better stream tracking and retry mechanism in Zenodo publication process
    The startZenodoPublishProcess function has been enhanced to track streaming status and implement a retry process if required. A new pending status has also been introduced to track uploads.

  • User feedback for pending uploads
    New user interface elements and messages have been added to provide feedback to users about the status of their pending uploads.

  • Heartbeat mechanism in post index
    A heartbeat mechanism has been added to keep the connection alive during prolonged steps in the Zenodo publication process.

  • Prevent indefinite hangs during network issues
    Request timeouts have been added to various API calls in zenodo.ts to prevent possible indefinite hangs during any network issues.

  • Consolidated error handling
    Error handling during Zenodo publications has been streamlined by consolidating it into a reusable failPublication function.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue, and left some high level feedback:

  • In reconcilePublishStatus, repeated failures to fetch /release/zenodo/status are silently ignored; consider logging or surfacing a more explicit error state so unexpected backend issues don’t leave the UI stuck in the generic pending state.
  • The fixed timeout values ZENODO_API_TIMEOUT_MS and ZENODO_UPLOAD_TIMEOUT_MS are hard-coded; consider making them configurable via environment variables so they can be tuned for different deployment environments or unusually large releases.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `reconcilePublishStatus`, repeated failures to fetch `/release/zenodo/status` are silently ignored; consider logging or surfacing a more explicit error state so unexpected backend issues don’t leave the UI stuck in the generic `pending` state.
- The fixed timeout values `ZENODO_API_TIMEOUT_MS` and `ZENODO_UPLOAD_TIMEOUT_MS` are hard-coded; consider making them configurable via environment variables so they can be tuned for different deployment environments or unusually large releases.

## Individual Comments

### Comment 1
<location path="ui/server/services/archival/zenodo.ts" line_range="579" />
<code_context>
       "Content-Type": "application/octet-stream",
     },
     method: "PUT",
+    signal: AbortSignal.timeout(ZENODO_UPLOAD_TIMEOUT_MS),
   });

</code_context>
<issue_to_address>
**suggestion:** Consider handling fetch timeouts explicitly so users see a clearer error than a generic AbortError message.

`AbortSignal.timeout` prevents hanging, but when it fires `fetch` rejects with an `AbortError` whose message (e.g. "The operation was aborted") is not very helpful. Since `failPublication` displays `err.message`, users will see a vague error. Consider catching `AbortError` around these long-running requests (or in a shared wrapper) and rethrowing a clearer message like "Zenodo upload timed out after 5 minutes; please retry" so timeouts are distinguishable from other failures.

Suggested implementation:

```typescript

/**
    body: body.toString(),
=======
 // Request timeouts so a stalled network call can never hang the publication
 // workflow indefinitely. Metadata/API calls are quick; file up/downloads and
 // the repo zipball can be large, so they get a much longer budget.
 const ZENODO_API_TIMEOUT_MS = 60_000; // metadata / publish / deposition / API calls
 const ZENODO_UPLOAD_TIMEOUT_MS = 300_000; // file up/downloads & repo zipball

 // ===== Error helper ==================================================

 /**
     body: body.toString(),
+
+/**
+ * Normalize errors from Zenodo calls so we can surface clearer messages to the user.
+ *
+ * In particular, when AbortSignal-based timeouts fire, fetch rejects with an AbortError
+ * whose default message ("The operation was aborted") is not helpful. Callers should
+ * pass the timeout and a short context string (e.g. "file upload") so we can generate
+ * a clearer message.
+ */
+export function normalizeZenodoError(
+  err: unknown,
+  timeoutMs?: number,
+  context?: string,
+): Error {
+  if (err instanceof Error && err.name === "AbortError" && timeoutMs && context) {
+    const minutes = Math.round(timeoutMs / 60_000);
+    return new Error(
+      `Zenodo ${context} timed out after ${minutes} minute${minutes === 1 ? "" : "s"}; please retry.`,
+    );
+  }
+
+  return err instanceof Error ? err : new Error("Zenodo request failed");
+}

```

To fully implement the suggestion, you should also:

1. **Wrap long-running fetches in try/catch and use `normalizeZenodoError`**. For example, where you perform metadata/API calls:
   - Locate the `fetch` that uses `signal: AbortSignal.timeout(ZENODO_API_TIMEOUT_MS)` (the one that produces `res` just above the `if (!res.ok)` block).
   - Wrap it as:
   ```ts
   let res: Response;
   try {
     res = await fetch(url, {
       // ...existing options...
       signal: AbortSignal.timeout(ZENODO_API_TIMEOUT_MS),
     });
   } catch (err) {
     throw normalizeZenodoError(err, ZENODO_API_TIMEOUT_MS, "API request");
   }
   ```
2. **Do the same for file uploads/downloads** using `ZENODO_UPLOAD_TIMEOUT_MS`:
   ```ts
   let res: Response;
   try {
     res = await fetch(uploadUrl, {
       // ...existing options...
       signal: AbortSignal.timeout(ZENODO_UPLOAD_TIMEOUT_MS),
     });
   } catch (err) {
     throw normalizeZenodoError(err, ZENODO_UPLOAD_TIMEOUT_MS, "file upload");
   }
   ```
3. **Ensure `failPublication` (or any shared error handler) still displays `err.message`**, but now the error has already been normalized by `normalizeZenodoError`, so users will see the clearer timeout-specific message instead of the generic `"The operation was aborted"`.

You may need to adjust import paths if `normalizeZenodoError` is moved to or used from other modules in your codebase.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

"Content-Type": "application/octet-stream",
},
method: "PUT",
signal: AbortSignal.timeout(ZENODO_UPLOAD_TIMEOUT_MS),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Consider handling fetch timeouts explicitly so users see a clearer error than a generic AbortError message.

AbortSignal.timeout prevents hanging, but when it fires fetch rejects with an AbortError whose message (e.g. "The operation was aborted") is not very helpful. Since failPublication displays err.message, users will see a vague error. Consider catching AbortError around these long-running requests (or in a shared wrapper) and rethrowing a clearer message like "Zenodo upload timed out after 5 minutes; please retry" so timeouts are distinguishable from other failures.

Suggested implementation:

/**
    body: body.toString(),
=======
 // Request timeouts so a stalled network call can never hang the publication
 // workflow indefinitely. Metadata/API calls are quick; file up/downloads and
 // the repo zipball can be large, so they get a much longer budget.
 const ZENODO_API_TIMEOUT_MS = 60_000; // metadata / publish / deposition / API calls
 const ZENODO_UPLOAD_TIMEOUT_MS = 300_000; // file up/downloads & repo zipball

 // ===== Error helper ==================================================

 /**
     body: body.toString(),
+
+/**
+ * Normalize errors from Zenodo calls so we can surface clearer messages to the user.
+ *
+ * In particular, when AbortSignal-based timeouts fire, fetch rejects with an AbortError
+ * whose default message ("The operation was aborted") is not helpful. Callers should
+ * pass the timeout and a short context string (e.g. "file upload") so we can generate
+ * a clearer message.
+ */
+export function normalizeZenodoError(
+  err: unknown,
+  timeoutMs?: number,
+  context?: string,
+): Error {
+  if (err instanceof Error && err.name === "AbortError" && timeoutMs && context) {
+    const minutes = Math.round(timeoutMs / 60_000);
+    return new Error(
+      `Zenodo ${context} timed out after ${minutes} minute${minutes === 1 ? "" : "s"}; please retry.`,
+    );
+  }
+
+  return err instanceof Error ? err : new Error("Zenodo request failed");
+}

To fully implement the suggestion, you should also:

  1. Wrap long-running fetches in try/catch and use normalizeZenodoError. For example, where you perform metadata/API calls:
    • Locate the fetch that uses signal: AbortSignal.timeout(ZENODO_API_TIMEOUT_MS) (the one that produces res just above the if (!res.ok) block).
    • Wrap it as:
    let res: Response;
    try {
      res = await fetch(url, {
        // ...existing options...
        signal: AbortSignal.timeout(ZENODO_API_TIMEOUT_MS),
      });
    } catch (err) {
      throw normalizeZenodoError(err, ZENODO_API_TIMEOUT_MS, "API request");
    }
  2. Do the same for file uploads/downloads using ZENODO_UPLOAD_TIMEOUT_MS:
    let res: Response;
    try {
      res = await fetch(uploadUrl, {
        // ...existing options...
        signal: AbortSignal.timeout(ZENODO_UPLOAD_TIMEOUT_MS),
      });
    } catch (err) {
      throw normalizeZenodoError(err, ZENODO_UPLOAD_TIMEOUT_MS, "file upload");
    }
  3. Ensure failPublication (or any shared error handler) still displays err.message, but now the error has already been normalized by normalizeZenodoError, so users will see the clearer timeout-specific message instead of the generic "The operation was aborted".

You may need to adjust import paths if normalizeZenodoError is moved to or used from other modules in your codebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant